Cloud computing data centers grow constantly in terms of size and capabilities. This includes the number of computing nodes. Computing nodes may be seen as one of three elements of a data center: nodes, switches and network connections. Additionally, routing protocols need to be taken into account. Today, cloud data centers (DC) host many thousands of active nodes, with a trend to go to tens of thousands nodes per data center. Each node can be a server (single, SMP [symmetrical multi-processing], small cluster), storage or an accelerator. All those nodes need to be interconnected via one (or several) partial DC networks. Many network topologies are known and well understood today: Many versions of tree structures (all-to-all (aka “full mesh”), 2D/3D mesh2D/3D torus, tree, fat tree, XGFT, dragonfly, Dragonfly+, 2 Tier dragonfly, Hamming graph, HyperX and many more), each with one or several network levels. Generally, for cloud computing data centers there are some basic goals: increasing the number of inter-connected nodes, having performance-efficient routing protocols and achieving the above goals while meeting important boundary conditions like costs, latency and bandwidth (i.e., performance) as well as RAS features (reliability, availability and serviceability).
However, each network topology has the same fundamental issue: Given the number of nodes, physical space requirements of the nodes dictate that nodes will be spaced apart by more than what an electrical cable at high speed (10+ Gbps) can bridge (this is given by laws of physics). As a result, many optical cables will be required in addition to network switches to implement one of the above-mentioned network topologies. One (eventually extreme) example: For a 1024 node all-to-all network connection, 500,000 optical cables would be required. If those are 100 Gbps Ethernet connections each, at current street-price of ˜$1,500/cable, this adds about $785 M to the cost of a DC, or $767 k per node! Hence, there is a need for a network structure which minimizes the number of optical (distance>˜5 m) cables. More generally: traditional network topologies can address the node scaling aspect of cloud computing data centers, but it is extremely difficult to simultaneously optimize also other boundary conditions.
There are several disclosures related to data center network structures. Document US 2010/002-0806 A1 discloses a method comprising receiving a data packet at a switch coupled to a plurality of switches. The switch may determine, from a first level table comprising a plurality of prefixes, a prefix matching a first portion of a destination address of the received packet. The switch may also determine, from a second level table comprising a plurality of suffixes, a suffix matching a second portion of a destination address of the received packet when the matching prefix of the first level table is associated with the second level table. This structure is used for an implementation of a data center communication fabric providing scalable communication bandwidth with significant fault tolerance using a plurality of smaller individual switches of substantially the same size and capacity.
Document U.S. Pat. No. 9,166,886 B1 discloses a system for determining a physical topology of the network including a plurality of nodes. Each node includes a multi-chip higher-tier switch. Each chip in the multi-chip higher-tier switch includes a plurality of ports. A network configuration module is configured to assign physical connections to respective ports of chips of the higher-tier switches through an iterative process. The iterative process includes selecting a first route of a plurality of routes, is signing for the source node of the selected route, a first portion on a first chip having an odd number of free ports.
One disadvantage of known solutions lies in the fact that an optimization of the underlying network characteristics is mainly made according only to one as the possible variables like selected network architecture, number of ports per switch, network dimension (2D, 3D, nD), reach of one hop connection, number of total required hops, and so on.
However, there may be a need to overcome this limitation of an optimization in only one direction. This may apply in particular to an optimization reflecting the above-mentioned boundaries as well as related costs for short distance and long-distance network connections in a large data center comprising hundreds or thousands of nodes.